Intra-Lingual and Cross-Lingual Prosody Modelling

Statistical Parametric Speech Synthesis (SPSS) offers flexibility and computational advantage compared to other methods for Text-to-Speech Synthesis. While the speech output is intelligible, statistically trained voices are less natural due to the amount of signal processing and statistical averaging that goes into building the models. Much of the blame for the lack of naturalness falls on the inappropriate and monotonous prosody in synthesized speech. The voice source, which directly effects the prosody, is a complementary source of information than the vocal tract and has its own patterns that need to be dealt with appropriately. Under this hypothesis, this thesis investigates the representations and optimal strategies for prosody modelling within the SPSS paradigm. We propose the Statistical Phrase/Accent Model (SPAM) of intonation as a framework that is both (i) a computational model with associated training and synthesis methods for prosody and (ii) has strong theoretical basis for prosodic description. The SPAM framework combines the strengths of existing complementary views of intonation like Autosegmental Metrical Phonology, Production paradigms like the Fujisaki Model and purely computational approaches like the TILT model. We demonstrate Accent Groups, a new data derived phonological unit, as the optimal representational level to model Pitch accents and integrate it within a multi-tier phonological model to synthesize natural and expressive intonation contours. In addition to improving text-to-speech synthesis, the framework is shown to improve voice conversion, both intra-lingually across speakers, and cross-lingually across languages. We apply the proposed techniques on synthesis of Audiobooks by incorporating richer semantic and contextual features beyond the sentence. We also look at the closely related problem of voice conversion within the SPAM framework to more effectively capture the speaking style of a target speaker. The techniques are also applied for the case of cross-lingual voice conversion, in the context of speech-to-speech machine translation which aims to automatically dub a video into a target language, while preserving the speaker’s intent in the original language after translation. Appropriate objective and subjective evaluations are conducted to show the performance of the proposed techniques.